Remembering the future : towards an application of genetic co-evolution in music improvisation

Musical improvisation is driven mainly by the unconscious mind, engaging the dialogic imagination to reference the entire cultural heritage of an improvisor in a single flash. This paper introduces a case study of evolutionary computation techniques, in particular genetic coevolution, as applied to the frequency domain using MPEG7 techniques, in order to create an artificial agent that mediates between an improvisor and her unconscious mind, to probe and unblock improvisatory action in live music performance or practice. . . . 1 Demons Versus Bounded Rationality “Composing is a slowed-down improvisation; often one cannot write fast enough to keep up with the stream of ideas.”Arnold Schoenberg, “Brahms the Progressive”, 1933, in Style and idea, 1950, as quoted in Nachmanovich, 1990. In the fields of cognitive science and economics, a raging battle is being fought, between those who believe human beings to be essentially rational beings, driven and informed by their preferences, endlessly calculating the probabilities for success of one decision over another, and those who believe human reasoning to operate within the bounds of the ecological reality humans face: limited time and information; and that we use fast and frugal heuristics to make most of our decisions. Gigerenzer and Todd’s [1] separation between Demons and Bounded Rationality is a clear example of this battlefront, and serves well to situate our point of departure for this paper. We believe that the processes behind musical improvisation, and therefore to a great extent those of composition, are not the result of an unbounded rationality at work, empowered solely by reasoning power, experience and musical training (Demons), but are more intrinsic, frugal and driven by a bounded rationality, influenced and sometimes entirely driven by the unconscious. 2 Remembering the Future In perhaps the most insightful book on improvisation we have read, (perhaps aside from Herrigel’s ‘Zen in the Art of Archery’ [2]), ‘Free Play’ Stephen Nachmanovitch [3] writes: “Intuition is a synaptic summation, our whole nervous system balancing and combining multivariate complexities in a single flash. It’s like computation; but while computation is a lineal process, going from A to B to C, intuition computes concentrically. All the steps and variables converge on the central decision-point at once, which is the present moment.” While we believe improvisation to be a bounded rationality process, we don’t necessarily agree that computation has to be a lineal process, and that certain genetic algorithm techniques, mimetic agencies, fast and frugal heuristics, and eventually emergent methods such as hierarchical temporal networks will allow us to create improvising entities in the computational domain to perhaps parallel those of human improvisors and certainly, help along the process of unblocking improvising skills for human improvisors, thus becoming helper, augmentative algorithms such as Frank aims to eventually be. We see successful free improvisors (Jarrett, Parker, Bailey, etc.) as performing an impossible feat : creating music compositions out of thin air, and on the spot. Sometimes, the feats they perform are so astounding, we cannot even recognise how it is done and must resort to calling it inspiration, or simply genius. But free improvisation is about listening and what Gladwell [4] calls ‘thin-slicing‘’, in that an expert improvisor is able to actively listen to her environment (other musicians, the room, the echoes in her memory) and ‘thin-slice‘ the content for clues she recognises as departure and arrival points, dialogic references and surprises, and then respond according to how her unconscious is directing her. Listening is a skill that can be acquired through training and matured through experience; so might thin-slicing, if one were able to control the environment in which an improvisation happens, and involve learning agents built specifically to unblock the unconscious. We propose to build such an agent, using methods inspired by Todd and Werner’s work on genetic co-evolution algorithms [5] and the ABC group’s theories on fast and frugal heuristics [1], as well as Michael Casey’s MPEG7 feature recognition techniques [6, 7] as implemented in his Soundspotter framework. Our work, needless to say, stands on the shoulder of giants. As well as Todd, Werner, Gigerenzer and Casey, we have benefited from the amazing vision of Thomas Grill, whose C++ framework for the Puredata environment allowed us to quickly prototype and think our way through our ideas with minimal programming pain, and from the amazing leaps of progress made by others, from Lewis‘ ‘Voyager’ [8] to Miranda‘s mimetic agents [9] and cellular automata systems. Our criteria for this agent are: – It must take input from live music improvisation as its main body of data and primary control device. Remembering the Future 3 – It must enable the player to navigate a map of unconscious musical gestures (musical phrases and their timbral, rhythmic interrelationships) by providing an evolving ‘mirror‘ to her playing. Many artificial agents have been built to provide independent and collaborative music improvisors, and we will outline a few that have influenced our research below; we will however firstly examine some of the further issues that have influenced the design of ours, whom we will call Frank, in honour of Todd and Werner’s Frankensteinian Methods. Note In our exposition of this system, we will refer many times to ‘sound gestures’. By this, and in their relationship to our ‘lexemes’ (which are nothing more than a captured set of moments in sound), we simply mean the timbral and rhythmic nature of a specific segment of sound in time, and we don’t at this point take in the larger, more involved concepts of sensori-motor control, large-scale timing and motivic returns. 1.1 Remembering the Future Improvisation happens in an environment full of snap judgments, where previous experience, cultural heritage and current information acquired through listening all help enable the improvisor to make decisions quickly. Snap judgments can be made in a snap because they are light in processing expense and frugal in nature [4, 1], and successful decision making in improvisation relies on a carefully nurtured balanced between bounded (deliberate) and unbounded (instinctive, unconscious) rationalities. In instinctive behavior, thin slices of experience are captured and processed by the unconscious to give us ready answers to questions which need an immediate answer, such as ‘If I don’t put my hand forward, will that door slam into me¿, or ‘Do I like this person enough to trust them with my child for 5 minutes?’, or ‘Is the violin player about to reference the motif I introduced 3 minutes ago, and should I join in¿. In the work of the improvisor, in her practice, there is an inescapable need to unblock unconscious action, so that these snap judgments can occur and meaningful musical material emerge. Improvisors such as Evan Parker rarely practice from a notated score, and choose instead to focus on gestural devices that have developed in their playing during decades of practice and live performance with others. His is then a self-contained ecology, where Lewis’ dialogic imagination [8] can work unencumbered by the (sometimes essential) constraints of the score, composer, player cycle; but, it relies heavily on an almost completely exploratory process and ecological reality, which takes decades to evolve to the mature point where the process is almost solely E-creative [10, 11]. Theater improvisation offers a perfect example of unblocking unconscious action as a necessary and essential process in the training of good improvisors: ‘ ‘In life, most of us are highly skilled at supressing action. All the improvisation teacher has to do is to reverse this skill and he creates very 4 Remembering the Future ‘gifted‘ improvisers. Bad improvisers block action, often with a high degree of skill. Good improvisers develop action.‘ ‘ Keith Johnstone, as quoted in Gladwell (2005) In trying to perform this reversal, we need the agent to be free from the traditional bounds of composition. As George Lewis points out [8]: “If we do not need to define improvised ways of producing knowledge as a subset of composition, then we can simply speak of an improvising machine as one that incorporates a dialogic imagination.” Frank tries to activate the dialogic processes of the improvisor’s mind, in particular the quicksilver heuristics involved in finding improvisational pathways within musical material through instrumental practice. Our aim is to enable a state of flow in the player, in which her dialogic imagination can be receptive to the kind of motivic/harmonic play mature Jazz musicians experience. Behind any unconscious action, there is encyclopedic knowledge that we cannot necessarily access through willed action, and this points at an important issue: really skilled improvisors are able not just to recall on demand past events and current motivic/harmonic changes; they are also able to ‘remember‘ the future: they can project their imagination into future events. The essential process behind this kind of projection into time is typical prefrontal cortex activity: humans and some animals use it to predict whether a gap is too long to jump over, a challenger too fierce to fight, or a crossing to dangerous to attempt. We use our previous experience, and play the possible event (successful crossing or getting run over) in our minds. The combination of prefrontal simulation and experiential memory could be called an unconscious remembering or replay of an event which may (fight) or may not (flight) happen: this is why we call it remembering the future. Unconscious remembering, or noetic [12, 13] (to know that an event occurred without remembering) memory, is, we propose, at the heart of dialogic interplay in musical improvisation, and the design of our system will attempt to prod the human improvisor to better understand the temporal connections underlying this process. Creating a Door to the Unconscious Goldstein, Gigerenzer and Todd’s work [14, 1] on the recognition heuristic, the simplest of their fast and frugal heuristics, which proves that efficient decision-making does not need very large amounts of information and can also rely on lack of knowledge, can be linked to Jacoby’s unconscious recollection (noetic) as explained above. It is clear that in an environment where we are forced to act on unconscious data to make a decision, we will make links that simply are not, and have never been there; when pushed, we invent. We propose that simply giving a musician an ongoing evolutive stream of mirrored (feeding back and forth from human to agent) sound gestures could potentially trigger a frugal process of recognition, and the E-creative processes. Remembering the Future 5 These could in turn help to navigate her unconscious to focus and direct (deliberate thinking) improvisational and compositional processes. Through the same process (thin-slicing) that we follow when selecting fruit at a market or choosing a mate, she could select from incoming streams of music gestures, as though ‘shopping’ for her own bits of unconscious dialogic metadata. 1.2 Previous Methodologies Evolutionary computing has, by now, a long record of application in musical research; to date, it remains generally focused on either computer music or musical cognition concerns [15]. We will not address the whole background of this work here, but instead will focus on the technniques that inspired our work. Two excellent surveys and general inquiries into the use and general application of genetic algorithms in music (out of many others) are Gartland-Jones and Copley’s ‘The Suitability of Genetic Algorithms for Musical Composition‘ [10] and Burton and Vladimirova’s ‘Generation of Musical Sequences with Genetic Techniques‘ [16], both of which focus on methodologies (theirs and others) that attempt to use genetic algorithms to generate musical material. Some, such as Biles’ ‘GenJam‘ [17], working within restricted premises such as 8th-note derivation within strict Jazz timelines, others, such as the IndagoSonus system, attempt to bypass the fitness bottleneck through GUI-driven evolutionary targets. In the case of Todd and Werner’s co-evolution principle, work towards the generation of musical material based on populations of hopeful singers and critics co-evolving at the same time. In the case of Lewis‘ ‘Voyager‘, with its legacy of Forth programming, and rule-based structure, we see a competent improvisor, but one that is necessarily fixed within the numerical MIDI domain (as are most others), and not as able to capture the gestural nuances embedded in timbre variation that can occur within musical improvisation. We do not here have the space to outline each in turn. Todd and Werner’s genetic co-evolution algorithm became our choice of implementation for Frank, due to its emphasis on evolving criticism, an essential part of the thin-slicing machine (Frank) we wanted to build and of cultural heritage as a phenomenon. However, as pointed out by Miranda, Todd, and Kirby [18], within Todd’s co-evolution, which evolves hopeful male singers and female critics in parallel, there is a ‘puzzling fundamental question‘ which is left unaddressed: where do the expectations of the female critics come from? We will address this question in our system in a brute, fundamental way: by allowing the human improvisor to determine the scale of expectancy as a variable. Since the improvisor’s live input has a direct effect on the female genotype, this gets around the expectancy provenance. However, Eduardo Miranda [9] has attempted to evolve expectations such as these using a mimetic model, to ‘...demonstrate that small community of interactive distributed agents ...can evolve a shared repertoire of melodies (or tunes) from scratch...‘, and this points to a serious improvement over co-evolution. But as his focus remains notational (melodic), we chose to remain within the co-evolution method for now (which we found more generally applicable to other bodies of 6 Remembering the Future data, such as the FFT frames in MPEG7 analysis) and to try to address the expectation issue within our design. 2 Technical Implementation For the rest of this paper, we will refer to one particular use case of Frank, for consistency purposes. In this case, one human player at any instrument (in this case, piano) will be the live input, through normal analog to digital conversion feeding into the Puredata environment, within which we host the objects (written in C++, using Flext) that constitute our agent, Frank. The player is given a Puredata patch to control some of the facets of Frank, such as initial lexical database creation and starting the GA process. The Frank framework consists of the following elements, which feed into each other in sequence as the live sound input comes into Puredata: – MPEG7 feature extraction – Acoustic Lexemes database creation from clustered MPEG7 frames – Co-evolution GA, taking live sound, and two other variables as input – Audio repository, which can be static or built from live sound A high-level overview of Frank’s design and data flow can be seen in figure 1, which outlines the four steps above and shows where human input and reception happen.